Pneumonia readmission prevention

ABSTRACT

A decision support tool is provided for discharging a patient by predicting the probability of a patient&#39;s readmission with pneumonia based on information available prior to discharge. The information used to make the prediction may include labs, vitals, diagnoses, and medications from prior encounters and from the current encounter. At least some of this information may be used to compute one or more severity metrics for the patient, such as a cancer score, an epilepsy or seizure score, a pneumococcal pneumonia score, and an instability score, to be input into one or more prediction models. An ensemble of machine learning models may be applied to the patient information to generate a prediction of that patient being readmitted with pneumonia within a future time interval. Based on the prediction, one or more intervening actions may be initiated to reduce the probability of readmission.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of, and claims priority from, U.S.patent application Ser. No. 16/457,082, filed on Jun. 28, 2019, thecontents of which are hereby incorporated herein in its entirety byreference.

BACKGROUND

Pneumonia is an inflammation of the lungs that has a wide range ofcauses and that can have significant consequences on a patient's healthand the healthcare industry. For example, a study of a population ofpatients found an incidence of community-acquired pneumonia (i.e.,pneumonia contracted outside a healthcare facility) to be 18.3 per 1000people, with a mean hospital admission of 7.6 days and cost of $6,929,and 0.6% of admissions due to community-acquired pneumonia were found toresult in patient death. Due to the impact of pneumonia, it is importantto accurately and promptly diagnose pneumonia. Additionally, somepneumonia patients are at an increased risk of recurrences, and thus, itis important to take steps to reduce this risk. Some tools exist fordetermining the severity of pneumonia to determine whether a personshould be admitted as a patient, but these tools use a simple rubricthat does not take into account patient history or a more holistic viewof the patient's condition such that they cannot predict readmission dueto pneumonia. Further, existing solutions for predicting readmissionutilize patient information that is not available until the time ofdischarge or even later, limiting the ability to effectuate an actualreduction in the readmission risk through discharge planning.

SUMMARY

Systems, methods and computer-readable media are provided for predictingthe probability of a patient's readmission with pneumonia based oninformation available prior to discharge, utilizing the prediction toprovide decision support during discharge of the patient to reduce therisk of readmission. The information used to make the prediction that isavailable prior to discharge may include labs, vitals, diagnoses, andmedications from prior encounters and from the current encounter. Inexemplary embodiments, information that is available only at the time ofdischarge or after discharge is not used. At least some of thisinformation may be used to compute one or more severity metrics for thepatient to be input, with other patient information, into one or moreprediction models. The severity metrics may include a cancer score, anepilepsy or seizure score, a pneumococcal pneumonia score, and aninstability score. An ensemble of machine learning models is applied tothe patient information, with each model in the ensemble using adifferent set of features within the patient information. Each modelproduces a prediction, and the predictions of all models within theensemble are combined to generate a prediction of that patient beingreadmitted with pneumonia within a future time interval. Based on theprediction, one or more intervening actions may be initiated to reducethe probability of readmission. An intervening action may includemodifying discharge protocol for the patient based on the predictedprobability, including ordering additional testing, orderingmedications, providing resources to reduce pneumonia risk for thepatient after discharge, adjusting discharge instructions, andscheduling additional examinations or a follow-up appointment with acare provider.

In yet another embodiment, one or more machine learning models aretrained to generate the prediction that a patient to be discharged willbe readmitted with pneumonia within a future interval. Accordingly,reference information for a reference population may be received, and aplurality of features for predicting readmission may be selected usingthe reference information. The features may be selected by applying asequential modeling technique to a potential feature pool. In exemplaryaspects, the sequential modeling technique includes an adaptive GLMnetmade up of three models being applied successively to the potentialfeature pool. The potential feature pool may be created by dividing thereference information into categorical features and continuous features.Further, a first gradient tree model may be trained using categoricalfeatures, and a second gradient tree model may be trained usingcontinuous features. Potential categorical features are identified usingthe first gradient tree model, and potential continuous features areidentified using second gradient tree model. In exemplary aspects, thepotential categorical features, the potential continuous features, andone or more of the severity metrics are combined to create the potentialfeature pool.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is described in detail below with reference to theattached drawing figures, wherein:

FIGS. 1A and 1B depict aspects of an illustrative operating environmentsuitable for practicing an embodiment of the disclosure;

FIG. 2 depicts a flow diagram of a method for reducing pneumoniareadmissions, in accordance with an embodiment of the disclosure;

FIG. 3 depicts a table mapping medication and diagnoses features inputinto the ensemble of models, in accordance with an embodiment of thepresent disclosure;

FIG. 4 depicts a flow chart of a method for building a pneumoniareadmission predictor suitable for use within a decision support system,in accordance with an embodiment of the disclosure;

FIG. 5 depicts a flow chart of a method for selecting features for apneumonia readmission predictor suitable for use within a decisionsupport system, in accordance with an embodiment of the disclosure;

FIG. 6 depicts a table indicating a count of features in an embodimentof the disclosure actually reduced to practice;

FIG. 7 depicts a table of model coefficients in an embodiment of thedisclosure actually reduced to practice; and

FIGS. 8A-C depict graphical illustrations of the performance of apneumonia readmission predictor suitable for use within a decisionsupport system and actually reduced to practice.

DETAILED DESCRIPTION

The subject matter of the present invention is described withspecificity herein to meet statutory requirements. However, thedescription itself is not intended to limit the scope of this patent.Rather, the inventors have contemplated that the claimed subject mattermight also be embodied in other ways, to include different steps orcombinations of steps similar to the ones described in this document, inconjunction with other present or future technologies. Moreover,although the terms “step” and/or “block” may be used herein to connotedifferent elements of methods employed, the terms should not beinterpreted as implying any particular order among or between varioussteps herein disclosed unless and except when the order of individualsteps is explicitly described.

As one skilled in the art will appreciate, embodiments of the inventionmay be embodied as, among other things: a method, system, or set ofinstructions embodied on one or more computer-readable media.Accordingly, the embodiments may take the form of a hardware embodiment,a software embodiment, or an embodiment combining software and hardware.In one embodiment, the invention takes the form of a computer-programproduct that includes computer-usable instructions embodied on one ormore computer-readable media, as discussed further with respect to FIGS.1A-1B.

Accordingly, at a high level, this disclosure describes, among otherthings, methods and systems for providing decision support duringdischarge using a prediction of a probability that a patient will bereadmitted with pneumonia within a future time interval. In exemplaryembodiments, the prediction is generated using one or more machinelearning models with information received prior to discharge, such aslaboratory results, medication or procedure orders, and diagnoses. Thisinformation may include information from previous encounters or thecurrent encounter. At least some of the information available prior todischarge may be used to compute one or more severity metrics for thepatient. The severity metrics may include a cancer score, an epilepsy orseizure score, a pneumococcal pneumonia score, and an instability scoreand may be input into at least some of the one or more machine learningmodels to determine the probability of readmission. In some embodiments,the methods and systems may be implemented as a decision supportcomputer application or tool for discharge planning for a patient who isdiagnosed with and/or treated for pneumonia within the currentencounter. For instance, the predicted probability of readmission maytrigger one or more intervening actions to reduce the likelihood ofreadmission, including modifying discharge instructions for the patient,scheduling additional examinations or a follow-up visit, orderinglaboratory tests, or ordering one or more medications for the patient.

As previously explained, pneumonia is an inflammation of the lungs,caused primarily, but not exclusively, by infections. Typically, thisinflammation causes the alveoli in the lungs to fill with liquid, whichcan lead to coughing, chest pain, fever, and difficulty breathing.Because of its wide range of causes, pneumonia is typically categorizedbased on where it was contracted. For instance, hospital (orhealthcare)-associated pneumonia (HCAP) is contracted while the patientis in a medical facility, while community-acquired pneumonia (CAP) iscontracted in the community, outside of a healthcare setting. Thesignificance of this distinction is that HCAP is more likely to becaused by an antibiotic-resistant pathogen such as methicillin-resistantStaphylococcus aureus, while CAP has a more diverse (andantibiotic-susceptible) range of possible causes (and may not ever haveits actual cause recorded at all).

Pneumonia generally and CAP in particular can have significantconsequences on a patient's health, which is illustrated through thehealthcare costs due to increased patient stays. For example, the meanannual healthcare cost for patients without CAP is $3,783 while the costto individuals with CAP is $20,961. A study of a population of patientsfound an incidence of CAP to be 18.3 per 1000 people, with a meanhospital admission of 7.6 days and a cost of $6,929. Not only is CAPcostly to the patient and provider, 10.6% of CAP admissions led todeath. Due to the impact of pneumonia, the Centers for Medicare andMedicaid Services include pneumonia within their Readmissions ReductionProgram, which directly penalizes hospitals with excess readmissions dueto pneumonia.

To combat the problems with pneumonia, it is important to accurately andpromptly diagnose pneumonia. Some tools exist to assist in determiningthe severity of pneumonia to determine whether a person should beadmitted. For instance, the CURB-65 rubric is a fairly simple rubric inwhich severity is determined by a count of risk factors; patients getone point for each of the following risk factors (leading to a scorebetween 0 and 5): Confusion, Blood Urea Nitrogen value greater than 7mmol/L (19 mg/dL), respiratory rate of ≥30 breaths per minute, low bloodpressure (systolic blood pressure <90 mmHg or diastolic blood pressure≤60 mmHg), and age is over 65. Another exiting tool is the PneumoniaSeverity Index (PSI), which incorporates a wider range of diagnoses andlab/vital results, as well as assigning different weights to somefactors. Yet, PSI is still a checklist where each value gives a fixednumber of points, combined to create a final score. The simplisticapproach with these existing tools fails to take into account patienthistory or a more holistic view on patient conditions, which limits theaccuracy. Additionally, these existing tools are not ideal foridentifying patients who are likely to be readmitted as that informationis either not reliably available in the patient's electronic medicalrecords, such as respiratory rate and pleural effusion on an X-ray.

Accordingly, embodiments of the present disclosure aim to accuratelyidentify patients with pneumonia who are likely to readmit for pneumoniaduring a future time interval. This process may include predicting alikelihood of being readmitted with pneumonia within a future time usinginformation, such as labs, vitals, diagnoses, and medications availableprior to discharge. This information may include information from priorencounters and from the current encounter. As such, timely andappropriate intervention can be initiated before the patient isdischarged, thereby reducing the likelihood of readmission.Specifically, embodiments include receiving patient information for apatient diagnosed with pneumonia in the current encounter. Thisinformation received and used includes information that is availableprior to patient discharge. In exemplary embodiments, information thatis available after discharge is not used. An ensemble of machinelearning models is applied to the patient information, with each modelwithin the ensemble using a different set of features within the patientinformation. Each model produces a prediction, and the predictions ofall models within the ensemble are combined to generate a prediction ofthat patient being readmitted with pneumonia within a future timeinterval. Based on the prediction, one or more intervening actions maybe initiated to reduce the probability of readmission. An interveningaction may include transmitting an electronic notification of thepatient's predicted readmission to a remote user device, such as adevice associated with the healthcare provider or a device associatedwith the patient. The intervening action may also include orderingadditional testing, ordering medications, providing resources to reducepneumonia risk for the patient after discharge including modifyingdischarge instructions, and scheduling intervention with a care providersuch as an examination prior to discharge and/or a follow-up visit.

In yet another embodiment, one or more machine learning models aretrained to generate the prediction that a patient to be discharged willbe readmitted with pneumonia within a future interval. Accordingly,reference information for a reference population may be received, and aplurality of features for predicting readmission may be selected usingthe reference information. The features may be selected by applying asequential modeling technique to a potential feature pool. In exemplaryaspects, the sequential modeling technique includes an adaptive GLMnetmade up of three models being applied successively to the potentialfeature pool. The potential feature pool may be created by dividing thereference information into categorical features and continuous features.Further, a first gradient tree model may be trained using categoricalfeatures, and a second gradient tree model may be trained usingcontinuous features. Potential categorical features are identified usingthe first gradient tree model, and potential continuous features areidentified using second gradient tree model. In exemplary aspects, thepotential categorical features, the potential continuous features, andone or more of the severity metrics are combined to create the potentialfeature pool.

Further, a plurality of prediction models may be trained to predict alikelihood that a patient will be readmitted with pneumonia within afuture time interval based on the selected features. Each predictionmodel may be trained on a different set of features. The plurality ofprediction models may comprise an ensemble of models, such asgeneralized linear models, each providing a probability, and theprobabilities may be combined to predict the likelihood of readmission.

Referring now to the drawings generally and, more specifically,referring to FIG. 1A, an aspect of an operating environment 100 isprovided suitable for practicing an embodiment of this disclosure.Certain items in block-diagram form are shown more for being able toreference something consistent with the nature of a patent than to implythat a certain component is or is not part of a certain device.Similarly, although some items are depicted in the singular form, pluralitems are contemplated as well (e.g., what is shown as one data storemight really be multiple data-stores distributed across multiplelocations). But showing every variation of each item might obscureaspects of the invention. Thus, for readability, items are shown andreferenced in the singular (while fully contemplating, where applicable,the plural).

As shown in FIG. 1A, example operating environment 100 provides anaspect of a computerized system for compiling and/or running anembodiment of a computer-decision support tool for predicting pneumoniareadmission. Environment 100 includes one or more electronic healthrecord (EHR) systems, such as hospital EHR system 160, communicativelycoupled to network 175, which is communicatively coupled to computersystem 120. In some embodiments, components of environment 100 that areshown as distinct components may be embodied as part of or within othercomponents of environment 100. For example, EHR systems 160 may compriseone or more EHR systems, such as hospital EHR systems, healthinformation exchange EHR systems, ambulatory clinic EHR systems,psychiatry/neurology EHR systems. Such EHR systems may be implemented incomputer system 120. Similarly, EHR system 160 may perform functions fortwo or more of the EHR systems (not shown).

Network 175 may comprise the Internet, and/or one or more publicnetworks, private networks, other communications networks such as acellular network, or similar network for facilitating communicationamong devices connected through the network. In some embodiments,network 175 may be determined based on factors such as the source anddestination of the information communicated over network 175, the pathbetween the source and destination, or the nature of the information.For example, intra-organization or internal communication may use aprivate network or virtual private network (VPN). Moreover, in someembodiments, items shown as being communicatively coupled to network 175may be directly communicatively coupled to other items showncommunicatively coupled to network 175.

In some embodiments, operating environment 100 may include a firewall(not shown) between a first component and network 175. In suchembodiments, the firewall may reside on a second component locatedbetween the first component and network 175, such as on a server (notshown), or reside on another component within network 175, or may resideon or as part of the first component.

Embodiments of EHR system 160 include one or more data stores of healthrecords, which may be stored on storage 121, and may further include oneor more computers or servers that facilitate the storing and retrievalof health records. In some embodiments, EHR system 160 may beimplemented as a cloud-based platform or may be distributed acrossmultiple physical locations. EHR system 160 may further include recordsystems that store real-time or near real-time patient (or user)information, such as wearable, bedside, or in-home patient monitors, forexample. Although FIG. 1A depicts an exemplary EHR system 160 that maybe used for storing patient information, it is contemplated that anembodiment may also rely on decision support application 140 and/ormonitor 141 for storing and retrieving patient record information, suchas information acquired from monitor 141.

Example operating environment 100 further includes a provideruser/clinician interface 142 communicatively coupled through network 175to EHR system 160. Although environment 100 depicts an indirectcommunicative coupling between user/clinician interface 142 and EHRsystem 160 through network 175, it is contemplated that an embodiment ofuser/clinician interface 142 is communicatively coupled to EHR system160 directly. An embodiment of user/clinician interface 142 takes theform of a graphical user interface operated by a software application orset of applications (e.g., decision support application 140) on acomputing device. In an embodiment, the application includes thePowerChart® software manufactured by Cerner Corporation. In anembodiment, the application is a Web-based application or applet. Ahealthcare provider application may facilitate accessing and receivinginformation from a user or healthcare provider about a specific patientor set of patients for which the likelihood of being readmitted forpneumonia is predicted according to the embodiments presented herein.Embodiments of user/clinician interface 142 also facilitate accessingand receiving information from a user or healthcare provider about aspecific patient or population of patients including patient history;healthcare resource data; physiological variables (e.g., vital signs)measurements, time series, and predictions (including plotting ordisplaying the determined outcome and/or issuing an alert) describedherein; or other health-related information, and facilitates the displayof results, recommendations, or orders, for example. In an embodiment,user/clinician interface 142 also facilitates receiving orders, such asorders for more resources, from a user based on the results ofpredictions. User/clinician interface 142 may also be used for providingdiagnostic services or evaluation of the performance of variousembodiments.

An embodiment of decision support application 140 comprises a softwareapplication or set of applications (which may include programs,routines, functions, or computer-performed services) residing on aclient computing device, on one or more servers in the cloud, ordistributed in the cloud and on a client computing device such as apersonal computer, laptop, smartphone, tablet, mobile computing device,front-end terminals in communication with back-end computing systems orother computing device(s) such as computing system 120 described below.In an embodiment, decision support application 140 includes a Web-basedapplication or applet (or set of applications) usable to provide ormanage user services provided by an embodiment of the invention. Forexample, in an embodiment, decision support application 140 facilitatesprocessing, interpreting, accessing, storing, retrieving, andcommunicating information acquired from monitor 141, EHR system 160, orstorage 121, including predictions and condition evaluations determinedby embodiments of the invention as described herein. In an embodiment,decision support application 140 sends a recommendation or notification(such as an alarm or other indication) directly to user/clinicianinterface 142 through network 175. In an embodiment, application 140sends a maintenance indication to user/clinician interface 142. In someembodiments, application 140 includes or is incorporated into acomputerized decision support tool, as described herein. Further, someembodiments of application 140 utilize user/clinician interface 142. Forinstance, in one embodiment of application 140, an interface component,such as user/clinician interface 142, may be used to facilitate accessby a user (including a clinician/caregiver or patient) to functions orinformation on monitor 141, such as operational settings or parameters,user identification, user data stored on monitor 141, and diagnosticservices or firmware updates for monitor 141, for example.

In some embodiments, application 140 and/or interface 142 facilitatesaccessing and receiving information from a user or health care providerabout a specific patient, a set of patients, or a population accordingto the embodiments presented herein. Such information may includehistorical data; health care resource data; variables measurements, timeseries, and predictions (including plotting or displaying the determinedoutcome and/or issuing an alert) described herein; or otherhealth-related information. Application 140 and/or interface 142 alsofacilitates the display of results, recommendations, or orders, forexample. In an embodiment, application 140 also facilitates receivingorders, scheduling time with care providers (including follow upvisits), or queries from a user, based on the results of the forecastedpneumonia readmission risk, which may utilize user interface 142 in someembodiments.

Decision support application 140 may also be used for providingdiagnostic services or evaluation of the performance of variousembodiments. As shown in example environment 100, in one embodiment,decision support application 140, or the computer system on which itoperates, is communicatively coupled to monitor 141 via network 175. Inan embodiment, patient monitor 141 communicates directly (or via network175) to computer system 120 and/or user/clinician interface 142. In anembodiment, monitor 141 (sometimes referred to herein as anpatient-interface component) comprises one or more sensor componentsoperable to acquire clinical or physiological information about apatient, such as various types of physiological measurements,physiological variables, or similar clinical information associated witha particular physical or mental state of the patient. Such clinical orphysiological information may be acquired by monitor 141 periodically,continuously, as needed, or as they become available, and may berepresented as one or more time series of measured variables. It is alsocontemplated that the clinical or physiological information about apatient or population of patients, such as the monitored variables,patient demographics, patient history, and/or clinical narrativesregarding the patient, used according to the embodiment of the inventiondisclosed herein may be received from a patient's historical data in EHRsystem 160, or from human measurements, human observations, orautomatically determined by sensors in proximity to the patient.

An embodiment of monitor 141 stores user-derived data locally orcommunicates data over network 175 to be stored remotely. In anembodiment, decision support application 140, or the computer system itis operating on, is wirelessly communicatively coupled to monitor 141.Application 140 may also be embodied as a software application or appoperating on a user's mobile device, as described above. In anembodiment, application 140 and monitor 141 are functional components ofthe same device, such as a device comprising a sensor, application, anda user interface. In an embodiment, decision support application 140 isin communication with or resides on a computing system that is embodiedas a base station, which may also include functionality for chargingmonitor 141 or downloading information from monitor 141.

Example operating environment 100 further includes computer system 120,which may take the form of a server, which is communicatively coupledthrough network 175 to EHR system 160, and storage 121. Computer system120 comprises one or more processors operable to receive instructionsand process them accordingly and may be embodied as a single computingdevice or multiple computing devices communicatively coupled to eachother. In one embodiment, processing actions performed by computersystem 120 are distributed among multiple locations such as one or morelocal clients and one or more remote servers and may be distributedacross the other components of example operating environment 100. Forexample, a portion of computer system 120 may be embodied on monitor 141or the computer system supporting application 140 for performing signalconditioning of a measured patient variable. In one embodiment, computersystem 120 comprises one or more computing devices, such as a server,desktop computer, laptop, or tablet, cloud-computing device ordistributed computing architecture, a portable computing device such asa laptop, tablet, ultra-mobile PC, or a mobile phone.

Embodiments of computer system 120 include computer software stack 125,which, in some embodiments, operates in the cloud as a distributedsystem on a virtualization layer within computer system 120, andincludes operating system 129. Operating system 129 may be implementedas a platform in the cloud and is capable of hosting a number ofservices such as services 122, 124, 126, and 128, described furtherherein. Some embodiments of operating system 129 comprise a distributedadaptive agent operating system. Embodiments of services 122, 124, 126,and 128 run as a local or distributed stack in the cloud, on one or morepersonal computers or servers such as computer system 120, and/or acomputing device running interface 142 and/or decision supportapplication 140. In some embodiments, user/clinician interface 142and/or decision support application 140 operate in conjunction withsoftware stack 125.

In embodiments, model variables indexing service 122 provide servicesthat facilitate retrieving frequent itemsets, extracting databaserecords, and cleaning the values of variables in records. For example,service 122 may perform functions for synonymic discovery, indexing ormapping variables in records, or mapping disparate health systems'ontologies, such as determining that a particular medication frequencyof a first record system is the same as another record system. In someembodiments, model variables indexing service 122 may invoke computationservices 126. Predictive models service 124 is generally responsible forproviding one or more models for predicting pneumonia readmission asdescribed in connection to methods 200, 400, and 500 of FIGS. 2, 4, and5 , respectively.

Computation services 126 perform statistical software operations, suchas computing the transformed variable predictions, transferred features(such as log and log1p functions of features), and severity indices asdescribed herein. In an embodiment, computation services 126 andpredictive models service 124 include computer software services orcomputer program routines. Computation services 126 also may includenatural language processing services (not shown) such as Discern nCode™developed by Cerner Corporation, or similar services. In an embodiment,computation services 126 include the services or routines that may beembodied as one or more software agents or computer software routines.Computation services 126 also may include services or routines forutilizing performing sequential modeling using one or more models,including decision trees and logistic models, for predicting pneumoniareadmission, such as the models described in connection to FIGS. 2-8C.

In some embodiments, stack 125 includes file system or cloud-services128. Some embodiments of file system/cloud-services 128 may comprise anApache Hadoop and Hbase framework or similar frameworks operable forproviding a distributed file system and which, in some embodiments,provide access to cloud-based services such as those provided by CernerHealthe Intent®. Additionally, some embodiments of filesystem/cloud-services 128 or stack 125 may comprise one or more streamprocessing services (not shown). For example, such stream processingservices may be embodied using IBM InfoSphere stream processingplatform, Twitter Storm stream processing, Ptolemy or Kepler streamprocessing software, or similar complex event processing (CEP)platforms, frameworks, or services, which may include the use ofmultiple such stream processing services (in parallel, serially, oroperating independently). Some embodiments of the invention also may beused in conjunction with Cerner Millennium®, Cerner CareAware®(including CareAware iBus®), Cerner CareCompass®, or similar productsand services.

Example operating environment 100 also includes storage 121 (or datastore 121), which, in some embodiments, includes patient data for acandidate or target patient (or information for multiple patients),including raw and processed patient data; variables associated withpatient recommendations; recommendation knowledge base; recommendationrules; recommendations; recommendation update statistics; an operationaldata store, which stores events, frequent itemsets (such as “X oftenhappens with Y”, for example), and itemsets index information;association rulebases; agent libraries, solvers and solver libraries,and other similar information including data and computer-usableinstructions; patient-derived data; and healthcare provider information,for example. It is contemplated that the term “data” used hereinincludes any information that can be stored in a computer storage deviceor system, such as user-derived data, computer usable instructions,software applications, or other information. In some embodiments,storage 121 comprises data store(s) associated with EHR system 160.Further, although depicted as a single storage store, storage 121 maycomprise one or more data stores, or may be in the cloud.

Turning briefly to FIG. 1B, there is shown one example embodiment ofcomputing system 180 representative of a system architecture that issuitable for computer systems such as computer system 120. Computingsystem 180 includes a bus 182 that directly or indirectly couples thefollowing devices: memory 184, one or more processors 186, one or morepresentation components 188, input/output (I/O) ports 190, input/outputcomponents 192, radio 196, and an illustrative power supply 194. Bus 182represents what may be one or more busses (such as an address bus, databus, or combination thereof). Although the various blocks of FIG. 1A areshown with lines for the sake of clarity, in reality, delineatingvarious components is not so clear, and metaphorically, the lines wouldmore accurately be grey and fuzzy. For example, one may consider apresentation component, such as a display device, to be an I/Ocomponent. Also, processors have memory. As such, the diagram of FIG. 1Ais merely illustrative of an exemplary computing system that can be usedin connection with one or more embodiments of the present invention.Distinction is not made between such categories as “workstation,”“server,” “laptop,” “hand-held device,” etc., as all are contemplatedwithin the scope of FIG. 1A and reference to “computing system.”

Computing system 180 typically includes a variety of computer-readablemedia. Computer-readable media can be any available media that can beaccessed by computing system 180 and includes both volatile andnonvolatile media, and removable and non-removable media. By way ofexample, and not limitation, computer-readable media may comprisecomputer storage media and communication media. Computer storage mediaincludes both volatile and nonvolatile, removable and non-removablemedia implemented in any method or technology for storage of informationsuch as computer-readable instructions, data structures, program modulesor other data. Computer storage media includes, but is not limited to,RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM,digital versatile disks (DVD) or other optical disk storage, magneticcassettes, magnetic tape, magnetic disk storage or other magneticstorage devices, or any other medium which can be used to store thedesired information and which can be accessed by computing system 180.Computer storage media does not comprise signals per se. Communicationmedia typically embodies computer-readable instructions, datastructures, program modules or other data in a modulated data signalsuch as a carrier wave or other transport mechanism and includes anyinformation delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media includes wired media such as awired network or direct-wired connection, and wireless media such asacoustic, RF, infrared and other wireless media. Combinations of any ofthe above should also be included within the scope of computer-readablemedia.

Memory 184 includes computer storage media in the form of volatileand/or nonvolatile memory. The memory may be removable, non-removable,or a combination thereof. Exemplary hardware devices include solid-statememory, hard drives, optical-disc drives, etc. Computing system 180includes one or more processors that read data from various entitiessuch as memory 184 or I/O components 192. Presentation component(s) 188present data indications to a user or other device. Exemplarypresentation components include a display device, speaker, printingcomponent, vibrating component, etc.

In some embodiments, computing system 180 comprises radio(s) 196 thatfacilitates communication with a wireless-telecommunications network.Illustrative wireless telecommunications technologies include CDMA,GPRS, TDMA, GSM, and the like. Radio 196 may additionally oralternatively facilitate other types of wireless communicationsincluding Wi-Fi, WiMAX, LTE, or other VoIP communications. As can beappreciated, in various embodiments, radio 196 can be configured tosupport multiple technologies and/or multiple radios can be utilized tosupport multiple technologies.

I/O ports 190 allow computing system 180 to be logically coupled toother devices, including I/O components 192, some of which may be builtin. Illustrative components include a microphone, joystick, game pad,satellite dish, scanner, printer, wireless device, etc. The I/Ocomponents 192 may provide a natural user interface (NUI) that processesair gestures, voice, or other physiological inputs generated by a user.In some instances, inputs may be transmitted to an appropriate networkelement for further processing. An NUI may implement any combination ofspeech recognition, stylus recognition, facial recognition, biometricrecognition, gesture recognition both on screen and adjacent to thescreen, air gestures, head and eye tracking, and touch recognition (asdescribed in more detail below) associated with a display of thecomputing system 180. The computing system 180 may be equipped withdepth cameras, such as stereoscopic camera systems, infrared camerasystems, RGB camera systems, touchscreen technology, and combinations ofthese, for gesture detection and recognition. Additionally, thecomputing system 180 may be equipped with accelerometers or gyroscopesthat enable detection of motion.

The architecture depicted in FIG. 1B is provided as one example of anynumber of suitable computer architectures, such as computingarchitectures that support local, distributed, or cloud-based softwareplatforms, and are suitable for supporting computer system 120.

Returning to FIG. 1A, in some embodiments, computer system 120 is acomputing system made up of one or more computing devices. In someembodiments, computer system 120 includes one or more software agentsand, in an embodiment, includes an adaptive multi-agent operatingsystem, but it will be appreciated that computer system 120 may alsotake the form of an adaptive single agent system or a non-agent system.Computer system 120 may be a distributed computing system, a dataprocessing system, a centralized computing system, a single computersuch as a desktop or laptop computer or a networked computing system.

Decision-Support for Reducing Risk of Pneumonia Readmission

Turning now to FIG. 2 , an example embodiment of a method for reducingthe risk of or preventing pneumonia readmission is provided and isreferred to generally as method 200. In particular, example method 200utilizes an ensemble of machine learning models with patient informationavailable prior to discharge to predict a likelihood that the patientwill be readmitted with pneumonia within a future time interval andproviding an intervention corresponding to the predicted likelihood. Insome embodiments, method 200 is suitable for implementation as acomputer-performed decision support tool or application for providingcare to pneumonia patients or discharge planning for pneumonia patientsbased on a predicted readmission likelihood that is more accurate thanconventional technology would otherwise allow.

In accordance with method 200, at step 210, a plurality of patientinformation is received. The patient information may comprise currentpatient data, patient demographic data, and/or historical patient data.In exemplary aspects, current patient data includes data relating to thepatient's current encounter (e.g., the current admission into thehealthcare facility). The current encounter information may include adiagnosis of and/or treatment for pneumonia, which may becommunity-acquired, healthcare-associated, viral, bacterial, and thelike. During the current encounter, the patient may be diagnosed with ortreated for other conditions, such as asthma, chronic obstructivepulmonary disease, chronic ulcer of skin, lung cancer, neoplasms,gastrointestinal cancer, epilepsy, for example. Current patientinformation may further include medication orders and lab results fromthe current encounter.

Patient demographics may include age, sex, race, nationality,socioeconomic status, marital status, and/or employment status. Thisdata may further include the patient's insurance information, such asthe insurance provider and the type of plan. Further, historical patientdata may include the patient's medical history and/or family history.Historical patient data may also include information about the patient'spast encounters at the current healthcare facility or other facilities.In exemplary embodiments, historical patient data includes previousdiagnoses, medications, and lab results. In some embodiments, theinformation received is limited to a predetermined past time frame, suchas the past two years.

This patient information may be received from different sources. Forinstance, in one embodiment, all patient data is received at step 210from the patient's EMR. In other embodiments, data relating to thepatient's current condition and/or patient demographics may be receiveddirectly from a user, such as the patient or a care provider, inputtingsuch information into a user device. Some current patient data, such aspatient variable values, may be received from one or more sensors ormonitoring devices or directly from a laboratory running the laboratoryprocedures. Additionally, historical patient information may be receivedfrom the patient's EMR and/or from insurance claims data for thepatient. In an alternative embodiment, the patient's history may bereceived directly from the patient, such as during registration whenadmitted for the current encounter.

At step 220, the patient's likelihood of being readmitted with pneumoniawithin a future time interval is predicted using a plurality of machinelearning models on least some of the patient information that isavailable and received prior to patient discharge. In exemplaryembodiments, the future time interval is 30 days from the currentencounter such that it is predicted whether the patient will bereadmitted with pneumonia within 30 days of discharge in the currentencounter. The patient information may include lab results, vital signmeasurements, diagnoses, and medication history from the currentencounter as well as prior encounters. In exemplary aspects, the patientinformation used to predict pneumonia readmission is information is onlyinformation that is available prior to discharge. In other words,information that becomes available at the time of discharge of thecurrent encounter or after is not used.

The prediction for readmission is generated using an ensemble of modelsoperating on different sets of variables (referred to herein asfeatures). In some embodiments, the ensemble of models (or ensemblemodeling) is used to determine a composite prediction from a set ofmultiple, diverse models each created to predict an outcome orcontribute towards predicting an outcome. In some embodiments, eachmodel in the ensemble (which may also be referred to as a member of theensemble) comprises a different modeling algorithm and/or may begenerated using different training data sets. In exemplary aspects, forinstance, the predictor is an ensemble of ten machine learning models,such as generalized linear models, where each member model is createdwith (and thus applied to) to a different set of features within thepatient's data to output a predicted probability of the patient beingreadmitted with pneumonia. The prediction (or contribution) of eachmember model may be aggregated or combined to generate a compositeprediction (also referred to herein as an overall prediction) for theunseen data. For example, the overall prediction may be the mean of theprobabilities computed by all of the member models. In some embodiments,the individual prediction of each ensemble member is weighted equallywhen determining the overall, composite prediction. In anotherembodiment, the ensemble member model predictions may be assigneddifferent weights. In some embodiments, the weighting may be based onlearned feedback about prior predictions

The combinations of features used with the ensemble of models mayinclude features relating to diagnoses, medications, hospitalizationhistory, laboratory results, and severity metrics, for example. Thesefeatures are discussed in greater detail below. The ensemble may usedifferent combinations of these types of features. In one embodiment,for example, 33 different features were used with 10 models.

Features relating to diagnosis and/or medications may include featuresbased directly on diagnoses or medications. Relevant diagnoses mayinclude other upper respiratory infection, asthma, gout, chronicobstructive pulmonary disease, and relevant medications may includealbuterol, non-ionic iodinated contrast media, or viral vaccinesassociated with one or more of these diagnoses. The presence of adiagnosis or medication may be relevant in the current encounter only orin the patient's history, which may include the current encounter.Current medications, for example, include medications recorded as beingprescribed during the patient's current encounter, while historicalmedications include medications recorded at any encounter within a pasttime frame, such as two years.

In exemplary aspects, identification of these diagnoses/medicationfeatures are based on the patient EHR. For instance, for diagnoses, anontology mapping scheme may be used to map diagnostic codes to commonnomenclature. For medications, an ontology mapping scheme may be used tomap a medication in the EHR to a “drug classification” concept tocollect the generic name of the medication. In exemplary embodiments,features directly relating to diagnoses or medications are binary suchthat the feature is either present or absent from the patient record.FIG. 3 depicts a reference table 300 that is used in some embodiments tomap human and code names for various features input into the ensemble ofmodels.

In one embodiment, the following diagnoses/medication features are usedwithin the ensemble:

Other Upper Respiratory Infection (diagnosis), Current Encounter;

Non-Ionic Iodinated Contrast Media (medication), Current Encounter;

Viral Vaccines (medication), Current Encounter;

Chronic Ulcer of Skin (diagnosis), Past 2 Years including CurrentEncounter;

Gout and Other Crystal Arthropathies (diagnosis), Past 2 Years includingCurrent Encounter;

Menstrual Disorders (diagnosis), Past 2 Years including CurrentEncounter;

Asthma/COPD, Past 2 Years including Current Encounter, which may beidentified by:

Diagnosis in the CCS Category “Asthma”;

Diagnosis in the CCS Category “Chronic obstructive pulmonary disease andbronchiectasis”;

Medication with generic name “albuterol”;

Medication with generic name “albuterol-ipratropium”;

Medication with drug classification “anticholinergic bronchodilators”;

Medication with generic name “formoterol”;

Antidiarrheals (medication), Past Two Years including Current Encounter;and

Antineoplastic Hormones (medication), Past Two Years including CurrentEncounter.

Note that at least some of the features listed above and discussed below(including the severity metrics) include a reference to the past twoyears. Reference to the past two years means that information may comefrom previous encounters within a predetermined past time frame. In someexemplary embodiments, this past time frame is two years. However, it iscontemplated that other time frames, such as one year, three years, fiveyears, and the like may be used instead. As such, reference to the “pasttwo years” with respect to the features is a reference to apredetermined past time frame that may be longer or shorter than thepast two years.

Laboratory features with laboratory and clinical event values may alsobe used to predict pneumonia readmission. Laboratory features mayinclude measurements performed within the current encounter and/or overa previous time frame, such as the past two years. In some embodiments,labs performed within the last two hours of the current encounter areexcluded. The laboratory features may be in the form of a “last” value,a “max” value, a “min” value, a “median” value, and a “range”.

As used herein, the “last” value may refer to the most recent availablevalue for that laboratory procedure. The “max” value may refer to thehighest value found in a past time frame, such as one year. In someaspects, if there are no values in the past time frame but are valuesfrom before that, then the most recent value may be used as the “max”value. The “min” value may refer to the lowest value in a past timeframe, such as one year, or, in some aspects, the most recent value ifthere are no values in the relevant past time frame. The “median” valuemay refer to the middle-most value (or the mean of the two middle-mostvalues, if the count is even) of all values within the past time frame,such as one year. In some aspects, if no values are in the relevant pasttime frame, the most recent value may be used for the median. Lastly,the “range” value may refer to the difference between the max and minvalue computed as previously described. In some embodiments, a value fora lab feature may be imputed if the value is missing entirely. Further,some lab features may comprise transformed lab values, such as valuestransformed using log and log1p functions. In some embodiments, log andlog1p, as used herein, refer to the natural logarithm functions.Further, in exemplary aspects, the log1p function refers to a functionthat performs a log transform on the sum of a given feature value plusone, which enables use of the function of features that may have a zerovalue.

In one embodiment, the following laboratory features are used within theensemble:

Last Values

Albumin (Imputed value: 3.3 g/dL)

Sodium (Imputed value: 138 mmol/L)

Chloride (Imputed value: 102 mmol/L)

Hemoglobin (Imputed value without anemia: 12; Imputed value with anemia:10.2 g/dL)

Monocyte Percentage (Imputed value: 8%)

Log of RBC Distribution Width (CV) (Imputed value without anemia:ln(14.1%); Imputed value with anemia: ln(15.1%))

Log of Shock Index (in terms of ratio of heart rate to systolic bloodpressure) (Imputed value: ln(82 bpm/127 mmHg))

Max Values

Temperature (Imputed value: 100° F.)

Log of Blood Urea Nitrogen (BUN) (Imputed value without AKI or CKD:ln(21 mg/dL); Imputed value with AKI or CKD: ln(40 mg/dL))

Median Values

Temperature (Imputed value: 98.1° F.)

Range Values

Alanine Aminotransferase (Imputed value without liver disease: 7units/L; Imputed value with liver disease: 22 units/L)

International Normalized Ratio (for prothrombin time) (Imputed without ahistory of warfarin, embolisms, dysrhythmias, or heart valve disorders:0; Imputed with a history of warfarin, embolisms, dysrhythmias, or heartvalve disorders: 0.12)

Lymphocyte Percentage (Imputed without a history of anemia: 10%; Imputedwith a history of anemia: 16%)

Some of these values may be found directly in the patient's recordswhile other values may be calculated. For example, if the patient'srecord does not include a monocyte percentage but does have monocytecount and total white blood count, the monocyte percentage may becalculated from those other values.

Features may also be based on the patient's hospitalization history,such as the patient's length of stay, number of past inpatient andemergency encounters, and number of distinct diagnoses. In exemplaryembodiments, values associated with the patient's hospitalizationhistory are transformed using log1p function when used as featureswithin the ensemble. In one embodiment, the following hospitalizationhistory features are used within the ensemble:

Length of Stay

Log1p of length of stay, measured in hours

Past Encounter Counts

Log1p of count of distinct inpatient episodes within the past 365 days

Log1p of count of distinct emergency episodes within the past 365 days

Past Diagnosis/Medication Counts

Log1p of count of distinct diagnoses (as determined by a databasehierarchy, such as CCS) within past 30 days

Log1p of count of distinct diagnoses (e.g., by CCS Category) within past90 days

Log1p of count of distinct medications (e.g., by generic name) withinpast 90 days

Further, one or more severity metrics may be computed and used topredict the patient's likelihood of a pneumonia readmission. Theseseverity metrics may relate to comorbidities, specific conditions, labresults, and/or vital sign values. Unlike with the diagnosis/medicationfeatures, these features are not binary, and unlike the laboratoryfeatures, there is a limited set of possible values. Instead, thesemetrics are designed to incorporate certain warning signs indicating apatient has or will have a deteriorating condition into a feature thatcan be easily processed through a prediction model. In other words,these metrics summarize different signs and metrics into a singlefeature that a model can process more easily without losing thedistinction that more signs are worse than fewer. In exemplary aspects,severity metrics include one or more of a cancer score and aninstability score. In other embodiments, an epilepsy/seizure scoreand/or a pneumococcal pneumonia score may be used. The prediction mayfurther be based on one or more of a peptic ulcer score, a hypertensionscore, a mood/psychosis score, a blood thinner score, an asthma/COPDscore, and a neuropathy score.

These severity scores represent unconventional metrics for measuringpatient health and predicting pneumonia. Further, these unconventionalscores help to provide an accurate prediction using the ensemble ofmodels without the use of information typically received only at orafter discharge. In this way, these scores and their use within theprediction models to reduce pneumonia readmission are unconventional andimprove upon existing decision support tools. Further aspects of thepresent disclosure also utilize a Charlson score, a severity metricderived from the Charlson Comorbidity index, with one or more of theother severity scores.

A cancer score may indicate whether a patient has received a cancerdiagnosis within a relevant time frame, such as two years. As such,determining a cancer score for a patient may include identifying thepresence of one or more cancer diagnoses, assigning each diagnoses witha point level, and combining the assigned points. For instance, eachdiagnoses may be assigned one point, all points may be added together,and a cap (or maximum score) may be applied to obtain the cancer scorefor that patient. In exemplary aspects, the cancer score is capped atthree such that values above three after summing the points are replacedwith a score of three. In some aspects, only diagnoses within apredetermined time frame, such as two years including the currentencounter, are included in computing the cancer score. In an exampleembodiment, the following cancers or related treatments are eachassigned a point if identified in the patient's record: cancer ofbronchus, lung; secondary malignancies; other gastrointestinal cancer;cancer of lymphatic and hematopoietic tissue; neoplasms of anunspecified nature or uncertain behavior; maintenance chemotherapy; andradiotherapy.

An instability score may be determined based on values of the patient'slab results. These values may indicate a degree of instability ordeterioration of the patient. These lab result values may be determinedfrom the most recent value, whether the value is in the currentencounter or a past encounter. In alternative embodiments, the value isonly used if it is in the current encounter. In one embodiment, onepoint is assigned to the presence of each lab result value indicatedbelow:

Calcium <8.25 mg/dL

Hematocrit <34%

Albumin <3 g/dL

Bicarbonate ≥32 mmol/L

Chloride <100 mmol/L

Serum Creatinine ≥1.3 mg/dL

Lymphocyte Percentage <10%

Further, in exemplary embodiments, the instability score is capped at amaximum value of 5. In some embodiments, where a value for a lab resultis not found, 0 points is given, but 1 point may be imputed for thehematocrit value when the patient has anemia.

A pneumococcal pneumonia score may be determined by assigning a pointfor having each of: (i) a diagnosis of pneumonia due to Streptococcuspneumonia during the current encounter; (ii) a medication prescribedduring the current encounter relating to symptoms of pneumococcalpneumonia; and (ii) a medication of a related vaccine prescribed duringthe current encounter. The diagnosis of pneumonia due to Streptococcuspneumonia may be determined by a diagnosis defined by the ICD-9 orICD-10 categories. Medications relating to symptoms of pneumococcalpneumonia may include ceftriaxone, cefotaxime, ampicillin-sulbactam,azithromycin, or moxifloxacin. The vaccine may include eitherpneumococcal 13-valent conjugate vaccine,” “pneumococcal 23-valentvaccine (obsolete),” or “pneumococcal 23-polyvalent vaccine”. If apatient's records include multiple items within one of the threecategories above, only one point may be assigned for that category inaccordance with some embodiments of the invention. Additionally, thepoints may be summed and capped at a maximum pneumococcal pneumoniascore of two.

Further, an epilepsy/seizure score may be determined in a similarfashion. The epilepsy/seizure score may, thus, indicate whether thepatient has had a diagnoses of epilepsy or other condition associatedwith seizures and/or whether the patent has been prescribed anassociated medication. For instance, one point may be given to selectdiagnoses and medications relating to seizures and/or epilepsy, thepoints may be summed, and a cap may be applied. In one embodiment, onepoint is assigned for the presence of each of the following:

Diagnosis in group “Epilepsy; Convulsions” during the past two years(including current encounter)

Diagnosis in the group “Other connective tissue disease” during the pasttwo years (including current encounter)

Medication of “Skeletal Muscle Relaxants” (grouped by drugclassification) in medication history table during the past two years(including current encounter)

Medication of “phenobarbital” (grouped by generic name) or“Miscellaneous Anticonvulsants” (grouped by drug classification) inmedication history table during the past two years (including currentencounter)

Medication of “Miscellaneous Anticonvulsants” (grouped by drugclassification) prescribed during the current encounter

Medication of “diazepam” (grouped by generic name) during the past twoyears (including current encounter)

Where there is multiple items within one of the above diagnosis ormedication groups, only one point is given. In exemplary embodiments,the score is capped at 3.

As previously stated, one severity metric that may be used is referredto as the Charlson score derived from the conventional CharlsonComorbidity index. The Charlson score is obtained by applying weights toparticular conditions. For instance, a weight of one may be applied tochronic obstructive pulmonary disease, renal conditions like chronickidney disease or chronic nephrotic syndrome), rheumatoid arthritis, anddiabetes with complications. A weight of two may be applied to malignantneoplasms, mild liver disease, congestive heart failure, dementia, andparaplegia/paralytic syndromes. A weight of two may be applied to severeliver disease and HIV/AIDS, while a weight of six may be applied tometastatic neoplasms. Each condition may be identified via a codingalgorithm, such as ICD-9 and ICD-10, assigned a weight, and then addedinto a patient's Charlson score. The final value for the Charlson scoremay be transformed using the log1p function.

In embodiments utilizing a peptic ulcer score, the peptic ulcer scoremay be computed by assigning a point for the presence of each of thefollowing: (i) a diagnosis in the category “Gastroduodenal ulcer (excepthemorrhage)” during the current encounter; (ii) medication of “H2antagonists” (grouped by drug classification) prescribed or inmedication history during the past two years (including currentencounter); and (iii) medication of “proton pump inhibitors” (grouped bydrug classification) prescribed or in medication history during the pasttwo years (including current encounter). In some embodiments, if thereare multiple items within one of the above categories in the patient'selectronic record, a maximum of one point is assigned for each category.The total points may be summed, and a cap of two may be applied.

Similarly, a hypertension sore may be determined by adding the pointsassigned for the presence of each of the following:

diagnosis in the group “Hypertension” during the past two years(including current encounter);

any of the following medications (grouped by drug classification) duringthe past two years (including current encounter), with each medicationcounting separately (e.g., multiple points are given for multiplemedications):

angiotensin converting enzyme (ACE) inhibitors

calcium channel blocking agents

loop diuretics

beta blockers, cardioselective

beta blockers, non-cardioselective

vasodilators

vasopressors

In example embodiments, this hypertension score is capped at sevenpoints.

A mood/psychosis score may be determined by assigning one point for eachof the following medications (grouped by drug classification) during thepast two years (including the current encounter):

miscellaneous anxiolytics, sedatives and hypnotics

miscellaneous antipsychotic agents

phenylpiperazine antidepressants

The points may be summed together with a capped score of two points.

An asthma/COPD score may be determined by assigning one point for eachof the following diagnosis and medications relating to asthma or chronicobstructive pulmonary disease:

Diagnosis in the groups “Asthma” or “Chronic obstructive pulmonarydisease and bronchiectasis” during the past two years (including currentencounter)

Medication of “albuterol” or “albuterol-ipratropium” (grouped by genericname) prescribed or in medication history during the past two years(including current encounter)

Medication of “anticholinergic bronchodilators” (grouped by drugclassification) prescribed or in medication history during the past twoyears (including current encounter)

Medication of “glucocorticoids” (grouped by drug classification)prescribed or in medication history during the past two years (includingcurrent encounter)

In exemplary aspects, the sum of the points is capped at one point.

A blood thinners score may be determined by assigning one point for eachof the following medications (grouped by drug classification) in thepatient's medication history during the past two years (including thecurrent encounter):

platelet aggregation inhibitors

coumarins and indanediones

heparins

factor Xa inhibitors

The points may be summed together with a capped score of two points.

A neuropathy score may be determined by assigning one point for each ofthe following diagnoses and medications:

Diagnosis in the group “Other nervous system disorders” during the pasttwo years (including current encounter)

Medication of “gabapentin” or “pregabalin” (grouped by generic name)prescribed or in medication history during the past two years (includingcurrent encounter)

Medication of “carBAMazepine” or “OXcarbazepine” (grouped by genericname) prescribed or in medication history during the past two years(including current encounter)

Medication of “SSNRI antidepressants” (grouped by drug classification)prescribed or in medication history during the past the two years(including current encounter)

Medication of “tricyclic antidepressants” (grouped by drugclassification) prescribed or in medication history during the past twoyears (including current encounter)

Medication of “buPROPion” (grouped by generic name) prescribed or inmedication history during the past two years (including currentencounter)

In exemplary aspects, only one point is given when there are multipleitems within a particular bulleted group above. Additionally, the sum ofthe points for the neuropathy score is capped at three points.

Returning to method 200 of FIG. 2 , at step 230, one or more interveningactions may be initiated based on the predicted likelihood ofreadmission with pneumonia. In some embodiments, an intervening actionmay include emitting or otherwise electronically communicating arecommendation or notification to a caregiver responsible for thepatient's care, such as a physician or nurse. This notification may bepresented via a user/clinician interface (such as interface 142described in FIG. 1A). The notification may indicate the predictedprobability of the patient being readmitted with pneumonia within thefuture time interval and/or present instructions to not discharge thepatient or to discharge patient with particular discharge instructions.Additionally, some embodiments of step 230 may further include storingthe result of the pneumonia readmission prediction in an EHR associatedwith the patient and further may include providing the patient's EHR (orfacilitating access to the EHR) in the notification.

In example embodiments, an intervening action may include modifying thepatient's discharge protocol to one that is designed to reduce thelikelihood of readmission. The modified discharge protocol may includerequiring additional approval of discharge by a care provider (which mayrequire further examination by a care provider), providing dischargeinstructions tailored to reduce the risk of readmission due topneumonia, scheduling a follow up appointment with a care providerwithin a specified time from discharge, ordering additional testing, andprescribing medications designed to reduce the risk of pneumoniadeveloping. As such, an intervening action may include scheduling a timefor a care provider to see the patient prior to discharge or schedulinga follow-up appointment within a designated time period that is lessthan the time period (e.g., 30 days) for which the pneumonia readmissionis forecasted. An intervening action may also include electronicallyadding one or more documents with special discharge instructions to aqueue associated with the patient's record, which may include a queuedesignating documents for printing and/or providing to the patient. Oneor more care providers, such as a discharge nurse, may be notified ofthe additional documentation. Further, additional testing to confirm theincrease risk and medications may be ordered prior to patient'sdischarge.

One or more of these intervening actions may be performed byautomatically modifying computer code executed in a healthcare softwareprogram for treating the patient and/or discharging planning, therebytransforming the program at runtime. For example in one embodiment, themodification comprises modifying (or generating new) computerinstructions to be executed at runtime in the program, the modificationmay correspond to a change in a discharge procedure due to the predictedreadmission.

In some embodiments, the intervening actions may be initiatedautomatically when the probability of readmission satisfies a thresholdlevel (i.e., a threshold probability). The threshold level may varydepending on the action. For instance, a threshold probability may beset for providing additional discharge instructions such that, when thethreshold is satisfied, an action for providing additional dischargeinstructions is automatically initiated. A different threshold, such asa higher probability, may be set for initiating an exam by a careprovider to approve discharge or for ordering additional testing. Asused herein, satisfying a threshold may refer to meeting or exceeding athreshold value.

Building a Pneumonia Readmission Predictor

FIGS. 4-5 illustrate example processes for building a pneumoniareadmission predictor, including selecting features for use in thepredictor, are provided. As described in greater detail below withrespect to FIGS. 4 and 5 , a variety of machine learning models may beutilized in predicting pneumonia readmissions. For instance, predictionmodel(s) may be used to predict pneumonia readmission while featureselection models may be used to identify and select the features orfeature sets that are used with the prediction model(s). Further, thefeature selection models may include potential feature selection modelsand final feature selection models. Additionally, these models may usedifferent types of machine learning algorithms. For instance, a firsttype (such as a gradient boosted tree model) may be used to selectfeatures for a potential feature pool, a second type (such as anadaptive GLMnet) may be used for selecting final feature sets, and athird type (such as generalized linear models) such as may be used forgenerating the actual prediction.

Turning to FIG. 4 , a flow chart is depicted to illustrate an examplemethod 400 for building a pneumonia readmission predictor for providingdecision support through discharge planning of pneumonia patients. Atstep 410, reference patient data is received. The reference populationof patients includes patients discharged from a healthcare facilityafter being diagnosed with and/or treated for pneumonia. The patientdata includes information, such as diagnoses, medications, lab results,vital signs, that was available prior to discharge.

In exemplary aspects, some features may be removed from the referencepatient data as a pre-processing step. The removed features may includefeatures not available prior to the time of discharge the patient. Insome embodiments, the removed features further included: features tiedto specific hospitals, features tied to prior outpatient duration,medications used by a very small fraction of encounters, and binaryfeatures that are present only in 1% or fewer of encounters.

At step 420, the reference patient data may be divided into a number ofgroups, with each group representing a subset of patients with thereference population. In exemplary embodiments, the reference patientdata is divided into ten groups. At step 430, feature selection isperformed a number of times equal to the number of groups so that anumber of feature sets equal to the number of groups is created. Assuch, in exemplary embodiments, feature selection is performed tentimes. Each time feature selection is performed, it is performed on thenumber of groups minus one. For instance, where there are ten groups,feature selection is performed on nine groups with one group being heldout. The group being held out of feature selection changes each timefeature selection is performed. The process of feature selection isdetailed in FIG. 5 .

For each feature set that is selected in step 430, a machine learningmodel may be trained to predict a likelihood of pneumonia readmissionusing the selected feature set as indicated at step 440. In exemplaryaspects, the machine learning model is a generalized linear model. Forinstance, the models may be logistic regression models; however, it iscontemplated that other generalized linear models may be used.

At step 450, an ensemble of models is created from the plurality ofmachine learning models built for each feature set in step 430. Thisensemble may be used to predict pneumonia readmission risk as describedin step 220 with respect to FIG. 2 . As such, the models within theensemble may be run simultaneously on new patient data, and each modelmay generate a likelihood of a patient being readmitted with pneumoniawithin future time interval, such as 30 days. The predicted risk, whichmay be in the form of a probability, from each model may be combined todetermine an overall risk level or probability. For instance, theoverall risk for a target patient may be the average probability outputfrom the models within the ensemble. Further, the overall risk may beused to trigger intervening action, through a decision-support tool fordischarge planning for example, to reduce the risk of readmission.

Turning to FIG. 500 , an example method 500 for selecting the featuresto use in a pneumonia readmission predictor (e.g., an ensemble of aplurality of models) is depicted. In exemplary aspects, features forthis pneumonia readmission predictor are chosen in a sequential process,with each step removing some features and leaving others to be processedby following steps (or to remain in a final model). As illustratedthrough method 500, feature selection may comprise afeature-type-specific selection and a general selection. As such, atstep 510, features within the received reference patient data aredivided according to the type of feature. In exemplary aspects, asindicated in FIG. 5 , the data is divided into continuous features andcategorical features. Categorical features have only two or threepossible values. Example categorical features includes the presence orabsence of a medication or diagnosis. Continuous features are featureswith more than three possible values and often include lab results andvital signs. The features may be divided this way so that an initialselection model can be run while taking into account potential basis forcertain types of features. For instance, it may be desirable for agradient tree decision algorithm to be applied to these featuresseparately because such an algorithm is biased towards features withhigher cardinality.

Accordingly, for each type of feature, a machine learning model may bebuilt as illustrated at steps 520 a and 520 b. In exemplary aspects,each model built for the type-specific selection is an XGBoost (alsoreferred to as a boosted gradient tree) model. Prior to building themodels, randomized features of each type may be created by permuting thefeatures, or rearranging the values. For example, every continuousfeature may be permuted such that the features are shuffled to retainthe same values but are assigned to different instances. Additionally, afraction (such as 10%) of the categorical features, chosen at random,may be permuted. Each model may then be built with the regular featuresand permuted versions. For example, a first model (such as a firstXGBoost model) may be built with continuous features and the permutedversions, and a second model (such as a second XGBoost model) may bebuilt with categorical features and the permuted versions.

At steps 530 a and 530 b, continuous features and categorical features,respectively, may be identified for potential selection. The featuresmay be identified based on satisfying a threshold of importance forpredicting pneumonia readmission. In exemplary aspects, for instance,the importance (in terms of gain) of each feature may be calculated foreach of the two models. Permuted features may be sorted based on theirimportance (gain), and the 95th percentile value may be found. The 95thpercentile demarcates features with a gain that is higher (or moreimportant) than 95% of all of the permuted features. This calculation ofthe 95th percentile may be performed separately for the permutedcategorical features and the permuted continuous features such thatseparate 95th percentile values were found for each type of feature.These 95th percentile values may then be used as thresholds forselecting the non-permuted (i.e., original, real) features as potentialfeatures for use in the predictor. Accordingly, features with valuessatisfying the threshold may be selected. In exemplary aspects, thisstep means that any feature with a gain greater than or equal to these95th percentile values for their respective feature type (continuous vs.categorical) may be retained from the type-specific model.

Once continuous and categorical features are selected for potential use,sequential modeling may be applied to all potential features (continuousand categorical), as indicated at step 540. Any permuted features thatpassed the respective threshold may be removed such that the models areapplied only to actual features. In some embodiments, one or moreseverity indices are not part of the previous steps for featureselection. In this case, severity indices that have been withheld may beadded back to the pool of potential features at step 540. In otherembodiments, the severity indices are all part of the initial featureselection process in steps 510-530, but are added back into the pool ofpotential features even if these features did not satisfy theimportance/gain threshold.

As used herein, sequential modeling refers to applying a plurality ofmodels in a sequence such that the output of at least one model is usedin building a subsequent model. In exemplary embodiments, the sequentialmodeling of step 540 includes three models. A first model may be builtfor the categorical and continuous features selected in steps 530 a and530 b. The coefficients of the features from the first model may then beused to build a second model, and the coefficients for features from thesecond model may be used in building a third model. The featuresselected by the third model may be the final features used for thepredictive model, such as one of the GLM models in the ensemblediscussed with respect to FIG. 4 .

In some embodiments, the plurality of models used for the sequentialmodeling in feature selection is an adaptive GLMnet, where each model isa GLMnet. Because a general linear model is not as sensitive to thedistinction between categorical and continuous features compared to agradient tree model, these models may be run on both types of featurestogether. In applying the sequential models, coefficients from eachmodel may be used to create a vector of penalty factors that are usedfor building the subsequent GLMnet. In one embodiment actually reducedto practice, the coefficients were selected by finding the lambda valuethat gives the minimum mean cross-validated error and then selecting thelargest lambda that has an error within one standard error of theminimum.

Embodiment Actually Reduced to Practice

In an embodiment actually reduced to practice, the primary pieces ofsoftware used were the XGBoost and GLMnet libraries, both available fromthe CRAN collection of libraries for the R programming language andRStudio development environment. Additional packages included data.tableand magrittr (for data management), foreach and doParallel (forparallelizing code), Hmisc and ggplot (for exploratory data analysis),and caret (for identifying rare features).

In building the models, certain criteria was used identify indexepisodes to be used for the reference dataset. This criteria included:

Episode takes place in 2012 or later;

Patient is age 18 or above;

Has a Pneumonia diagnostic code and either:

The Pneumonia diagnosis has a priority of Primary; or

There exists a Sepsis diagnosis with a priority of Primary and theredoes not exist a Severe Sepsis diagnosis;

Admission source is not a Transfer;

Episode type is not one of the following:

Home Health/Hospice/LTC/SNF/Nursing Home;

Outpatient/Clinic/Day Surgery/Outpatient Surgery/Recurring; or

Non-Patient/Unknown-Invalid/Not Mapped

Discharge is not one of the following:

Expired/Hospice;

Null/Not Mapped/Unknown-Invalid;

Left Against Medical Advice;

Transfer to Law Enforcement; or

Transfer to Hospital;

Discharged at least one day after admission (e.g., discharge date minusadmission date ≥1);

Does not have a qualifying pneumonia encounter (e.g., one meeting theabove criteria) in the 30 days before the index;

At least one prior episode in the previous two years;

One of the following:

Has a follow-up episode between 30 and 395 days following index; or

Has an unplanned inpatient episode between 0 and 30 days following indexand the patient died or was discharged to hospice at the end of thefollow-up episode;

One of the following:

Does not have an unavoidable inpatient episode between 0 and 30 daysfollowing the index; or

Has an unplanned avoidable inpatient episode before having anunavoidable inpatient episode between 0 and 30 days following the index;

Comes from a hospital and month with one the following property:

For each hospital and month, consider all potential index episodes(those which passed the above criteria) that occurred during this monthand at this hospital;

For each of these index episodes, check if the episode(s) during theirtwo-year lookback window had at least one medication and at least oneprocedure;

Compute the fraction of all episodes for this hospital and month thathad at least one medication and at least one procedure; and

Use only those hospital-month pairings with at least 30 episodes, whereover 60% of those episodes had a medication during their lookback windowand over 50% of those episodes had a procedure during their lookbackwindow as valid pairings;

Comes from a hospital that was not excluded due to being an outlier withrespect to certain features;

Has a hemoglobin test within the current episode;

Has at least one blood urea nitrogen (BUN) test and one systolic bloodpressure (SBP) test within the past two years;

If SBP is missing but Diastolic Blood Pressure and Mean ArterialPressure are available, SBP is calculated from these and the episode maybe included;

The class value (outcome variable) was defined as follows:

There exists an unplanned inpatient episode (called the “follow-upepisode”) between 24 hours and 30 days following discharge from theindex episode.

Because embodiments uses an ensemble of ten different models trained onslightly different versions of the training data, not all of thefeatures discussed herein were used in every ensemble member. FIG. 6provides a chart 600 that indicates how frequently each feature appearedin the embodiment actually reduced to practice.

As stated, in some embodiments, the prediction is made using an ensembleof ten logistic regression models, each with their own modelcoefficients. FIG. 7 illustrates a table 700 that identifies theminimum, maximum, and median values for coefficients across ten modelsas well as whether or not the coefficient flipped sign in any of themodels in an embodiment actually reduced to practice. For all binaryfeatures, the coefficient in table 700 is the value added into the modelwhen the feature was present; if the feature was absent, that value wasnot added.

To evaluate the performance of the predictor actually reduced topractice, a 10-fold cross validation was applied, performing the entirefeature selection and model-building process (but not the choice ofimputation values for the laboratory features) inside thecross-validation. In other words, within each fold of the outer, testingcross-validation process, the entire model-building cross-validationprocess was performed. Averaging together the AUC (area under the curve)results for each run of testing cross-validation, the overall AUC was0.689. FIG. 8A depicts ROC (receiver operating characteristic) plots 800for all ten runs overlaid on top of each other. FIG. 8B depicts ahistogram and density plot 810 of all AUC values for the ten runs, whichillustrates the variation embodied by the different runs. FIG. 8Cillustrates a plot 820 validating the probabilities predicted by themodel, demonstrating how the predicted probabilities of each instancecompare with the empirical probabilities for instances from a givendegree of risk. As illustrated, embodiments of the disclosure improveupon conventional methods at least by yielding fairly accuratepredictions of pneumonia readmission prior to discharge such thatintervening action may be initiated and instituted to effectuate areduction in readmissions.

Many different arrangements of the various components depicted, as wellas components not shown, are possible without departing from the spiritand scope of the present invention. It will be understood that certainfeatures and subcombinations are of utility and may be employed withoutreference to other features and subcombinations and are contemplatedwithin the scope of the claims. Not all steps listed in the variousfigures need be carried out in the specific order described.Accordingly, the scope of the invention is intended to be limited onlyby the following claims.

What is claimed is:
 1. Computer storage media having computer-executable instructions embodied thereon that, when executed, provide a method for implementing a decision support tool for pneumonia patients, the method comprising: generating a prediction of a patient being readmitted for pneumonia; and based on the prediction of whether the patient will be readmitted for pneumonia, initiating one or more intervening actions prior to discharge to reduce a likelihood of readmission, the one or more intervening actions performed by automatically modifying computer code executed in a healthcare software program for treating the patient and/or discharging planning, thereby transforming the program at runtime.
 2. The computer storage media of claim 1, further comprising receiving patient information for a patient being treated for pneumonia.
 3. The computer storage media of claim 2, further comprising applying an ensemble of models to the patient information to generate the prediction of the patient being readmitted for pneumonia within a future time interval.
 4. The computer storage media of claim 3, wherein the patient information used to generate the prediction is available prior to discharge of the patient.
 5. The computer storage media of claim 1, further comprising receiving reference information on a reference population.
 6. The computer storage media of claim 5, further comprising selecting a plurality of features for predicting pneumonia readmisison, wherein selecting the plurality of features comprises applying sequential modeling using a plurality of feature-selection models to a potential feature pool based on reference information available prior to discharge.
 7. The computer storage media of claim 6, further comprising training a plurality of prediction models to predict a likelihood that a patient will be readmitted with pneumonia within a future time interval, each prediction model being trained on a different set of features from the plurality of features.
 8. The computer storage media of claim 7, further comprising generating a prediction that a target patient will be readmitted with pneumonia within the future time interval using the plurality of prediction models, wherein each prediction model provides a probability and wherein the probabilities from the plurality of prediction models are combined to generate the prediction, wherein the prediction is generated prior to discharge of the target patient.
 9. The computer storage media of claim 8, further comprising: dividing the reference information into categorical features and continuous features; and training a first gradient tree model using the categorical features and a second gradient tree model using the continuous features.
 10. The computer storage media of claim 0, further comprising: identifying potential categorical features from the first gradient tree model and potential continuous features from the second gradient tree model; and combining the potential categorical features, the potential continuous features, and one or more severity metrics to create the potential feature pool.
 11. The computer storage media of claim 10, wherein the plurality of prediction models comprises an ensemble of ten linear regression models and wherein the plurality of feature-selection models comprises an adaptive GLMnet, and further wherein the prediction that the target patient will be readmitted with pneumonia within the future time interval is used to automatically initiate one or more intervening actions to reduce the likelihood of readmission.
 12. A computer system for providing a discharge decision support tool for reducing readmissions due to pneumonia, the system comprising: one or more processors; computer storage media storing computer-useable instructions that, when executed by the one or more processors, implement a method comprising: generating a prediction of a patient being readmitted for pneumonia; and based on the prediction of whether the patient will be readmitted for pneumonia, initiating one or more intervening actions prior to discharge to reduce a likelihood of readmission, the one or more intervening actions performed by automatically modifying computer code executed in a healthcare software program for treating the patient and/or discharging planning, thereby transforming the program at runtime.
 12. The computer system of claim 11, further comprising receiving reference information on a reference population.
 13. The computer system of claim 12, further comprising selecting a plurality of features for predicting pneumonia readmisison, wherein selecting the plurality of features comprises applying sequential modeling using a plurality of feature-selection models to a potential feature pool based on reference information available prior to discharge.
 14. The computer system of claim 13, further comprising: training a plurality of prediction models to predict a likelihood that a patient will be readmitted with pneumonia within a future time interval, each prediction model being trained on a different set of features from the plurality of features; and generating a prediction that a target patient will be readmitted with pneumonia within the future time interval using the plurality of prediction models, wherein each prediction model provides a probability and wherein the probabilities from the plurality of prediction models are combined to generate the prediction, wherein the prediction is generated prior to discharge of the target patient.
 15. The computer system of claim 14, further comprising: dividing the reference information into categorical features and continuous features; training a first gradient tree model using the categorical features and a second gradient tree model using the continuous features; identifying potential categorical features from the first gradient tree model and potential continuous features from the second gradient tree model; and combining the potential categorical features, the potential continuous features, and one or more severity metrics to create the potential feature pool.
 16. The computer system of claim 11, wherein the plurality of prediction models comprises an ensemble of ten linear regression models and wherein the plurality of feature-selection models comprises an adaptive GLMnet, and further wherein the prediction that the target patient will be readmitted with pneumonia within the future time interval is used to automatically initiate one or more intervening actions to reduce the likelihood of readmission.
 17. A computerized method for providing a discharge decision support tool to reduce pneumonia readmissions, the method comprising: generating a prediction of a patient being readmitted for pneumonia; and based on the prediction of whether the patient will be readmitted for pneumonia, initiating one or more intervening actions prior to discharge to reduce a likelihood of readmission, the one or more intervening actions performed by automatically modifying computer code executed in a healthcare software program for treating the patient and/or discharging planning, thereby transforming the program at runtime.
 18. The computerized method of claim 17, further comprising: receiving reference information on a reference population; selecting a plurality of features for predicting pneumonia readmisison, wherein selecting the plurality of features comprises applying sequential modeling using a plurality of feature-selection models to a potential feature pool based on reference information available prior to discharge; training a plurality of prediction models to predict a likelihood that a patient will be readmitted with pneumonia within a future time interval, each prediction model being trained on a different set of features from the plurality of features; and generating a prediction that a target patient will be readmitted with pneumonia within the future time interval using the plurality of prediction models, wherein each prediction model provides a probability and wherein the probabilities from the plurality of prediction models are combined to generate the prediction, wherein the prediction is generated prior to discharge of the target patient.
 19. The computerized method of claim 18, wherein selecting the plurality of features further comprises determining the potential feature pool by: dividing the reference information into categorical features and continuous features; training a first gradient tree model using the categorical features and a second gradient tree model using the continuous features; identifying potential categorical features from the first gradient tree model and potential continuous features from the second gradient tree model; and combining the potential categorical features, the potential continuous features, and one or more severity metrics to create the potential feature pool.
 20. The computerized method of claim 19, wherein the plurality of prediction models comprises an ensemble of ten linear regression models and wherein the plurality of feature-selection models comprises an adaptive GLMnet, and further wherein the prediction that the target patient will be readmitted with pneumonia within the future time interval is used to automatically initiate one or more intervening actions to reduce the likelihood of readmission. 